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(54) A system for accessing multimedia mailboxes and messages over the internet and via 
telephone 



(57) A unified messaging system that provides a 
multimedia mailbox. The system allows a subscriber to 
access scored multimedia messages, such as voicemail 
messages, facsimile messages, combined voice and 
facsimile messages and video messages, not only 
through a public switched telephone network using a tel- 
ephone but also over a data network, such as the Inter- 
net or an intranet, using a personal computer. The sys- 
tem provides voicemail accessiover the telephone net- 
work, indicating message number, etc. with the ability to 
play messages to the telephone user as desired. For 
text type messages, such as facsimile and e-mail, the 
system converts the text into.; speech and plays the 



speech to the telephone user. The system allows a per- 
sonal computer user to obtain the data network access 
using an Internet browser. The browser is used to ac- 
cess a home page of the system and get information 
about the messages stored, and is used to download 
(get) and play the messages at the personal computer 
via data streaming in the case of a voice or video mes- 
sages or view the messages in the case of text type 
messages, such as facsimile and e-mail. The user can 
also perform the other typical messaging functions over 
the data network connection that are provided for tele- 
phone access, such as viewing a message list, saving 
and deleting messages, group list administration and 
other administration tasks. 
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Description 

REFERENCE TO MICROFICHE APPENDIX 

A microfiche appendix having 5 microfiche and 490 
frames is included herewith that includes source code 
in the C + 7HTML languages. 

BACKGROUND OF THE INVENTION 

Field of the invention 

The present invention is directed to a system for ac- 
cessing stored messages over a network and, more par- 
ticularly, is directed to a system for providing unified ac- 
cess to stored messages, such as multimedia mail mes- 
sages, in a unified multimedia mailbox through multiple 
access pathways such as over a telephone network us- 
ing a telephone and over the Internet using a browser 

Description of the Related Art 

Communication systems currently exist that allow 
different types of messages, such as voice mail mes- 
sages and facsimile messages, to be stored for later re- 
trieval by a subscriber to such systems. These types of 
systems are described in U.S. patents 5,029,199; 
5,193,110; 5,260,990; 5,263,080; 5,475,748; 
5,493,607; 5,524,139; 5,519,766 and 5,008,926, all in- 
corporated by reference herein. These systems allow a 
caller or sender to leave a message, such as a voice 
mail message, for a subscriber whenever the subscriber 
is not available. When a voice mail message is to be 
retrieved the subscriber typically connects with the sys- 
tem over a conventional telephone line via a telephone 
call and plays the message by using the touchtones pro- 
duced by the telephone to control playback, as well as 
other functions. In these systems the access by the sub- 
scriber is typically only through a telephone line connec- 
tion. Today, there is a need to allow access to such sys- 
tems through other means such as the Internet or In- 
tranet. 

Several different types of messaging systems, such 
as voice mail and e-mail, are also available to users. 
Users of the variety of today's messaging systems typ- 
ically have to use several different systems and/or ter- 
minals to get their messages. A typical business user 
may have severai voice mailboxes, several e-maii mail- 
boxes, and perhaps some mailbox-like facsimile servic- 
es. Each of these mailboxes requires separate opera- 
tions and different types of terminals (DTMF telephone 
for voice mail, personal computer (PC) for e-mail ac- 
cess, facsimile machine/PC for facsimile messages). 
The mailboxes have different names (addresses) and 
cannot usually interwork. Notification mechanisms are 
either non-existent, ortiedtooneof the mailboxes. What 
is needed is a mailbox system that integrates all of these 
message types and access methods. 



SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a 
system that allows a subscriber to access stored mes- 
5 sages over not only a telephone network but also over 
a network such as the Internet or an intranet. 

It is an additional object of the present invention to 
provide a system that unifies message storage allowing 
different types of messages or electronic communica- 
10 tions.such as voicemail, facsimile, e-mail and video mail 
to be stored on a single system in a single unified mul- 
timedia mailbox, and accessed via different pathways, 
such as via a telephone network or the Internet/Intranet. 

It is an object of the present invention to provide a 
system that will allow multimedia messaging via a mul- 
timedia mail box. 

It is another object of the present invention to pro- 
vide a system that is easy to use and which uses access 
tools that are familiar to telephone and Internet users. 

It is a further objeel of the presenl invention to pro- 
vide a simple visual interface to a message storage sys- 
tem that simplifies the tasks associated with message 
access and administration. 

It is also an object of the present invention to provide 
a platform that allows services for a variety of message 
types such as voice mail, video mail and facsimile mail, 
as well as other network services such as Internet and 
intranet services. 

It is another object of the present invention to pro- 
vide a system architecture that >s easily scaleable, has 
a high availability and which provides a fast response. 

It is an object of the present invention to provide a 
standards based system that will support mailbox ac- 
cess to a multimedia mailbox using conventional web 
browser software. 

It is another object of the present invention to pro- 
vide a system in which the message service provider 
does not need to supply the user with any client appli- 
cation software. 
40 it is a further object of the present invention to pro- 
vide message waiting/urgent notifiers when new or ur- 
gent messages are deposited in the mailbox or the mes- 
sage status changes by a simultaneous different con- 
nection into the mailbox such as when a mailbox is ac- 
45. cessed by computer and while the computer is logged 
into the mailbox an access via a telephone, interface de- 
letes a message. 

The above objects can be attained by a system that 
allows a subscriber to access stored messages, such 
so as voicemail messages, facsimile messages, e-mail 
messages and video messages, that are stored in a uni- 
fied multimedia mailbox not. only through a public 
switched telephone network using a telephone but also 
over a data network, such as the Internet or an intranet. 
55 The system provides voicemail access over the tele- 
phone network, indicating message number, etc. with 
the ability to play messages to !he telephone user. For 
text type messages, such as facsimile and e-mail, the 
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system converts the text into i speech and plays the 
speech to the telephone user. The system allows a per- 
sonal computer user to obtain the data network access 
using an Internet browser. The browser is used to ac- 
cess information about the messages stored and is used 5 
to download and play the messages via data streaming 
in the case of a voice or video, messages or view the 
messages in the case of text type messages, such as 
facsimile and e-mail. The user can also perform the oth- 
er typical messaging functions; over the data network io 
connection that are provided fontelephone access, such 
as saving and deleting messages, group list administra- 
tion and other administration tasks 

These together with other objects and advantages 
which will be subsequently apparent, reside in the de- is 
tails of construction and operation as more fully herein- 
after described and claimed, reference being had to the 
accompanying drawings forming a part hereof, wherein 
like numerals refer to like parts,throughoul. 

i 20 
BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a virtual unified system unified 
by the actions of a personal computer 60; 
Figure 2 depicts a unified virtual system unified by 25 
director 80; 

Figure 3 depicts a voice mail system 66 performing 
unification; 

Figure 4 depicts real integrated system 100; 
Figure 5 depicts the functional architecture of a sys- 30 
tern according to the present invention; 
Figure 6 depicts a distributed architecture system 
130 according to the present invention; 
Figure 7 illustrates the flow of control during a re- 
fresh operation; 35 
Figures 8 and 9 depict message list and group list 
templates, respectively; ' 

Figure 10 illustrates the flow of control in a retrieval 
operation; and - 

Figure 11 depicts a message retrieval template. 40 



DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The present invention provides an integrated multi- *s 
media mailbox and unified messaging. The term "mail- 
box" is used to mean an entity visible to the subscriber. 
This is the entity the subscriber logs into and appears 
to operate on when the subscriber performs mail-related 
operations. This subscriber-visible entity may not corre- so 
spond directly with a single implementation entity, but 
may exist only through the cooperation of several dis- 
tinct messaging systems, each wich it's own message 
storage capability. To avoid confusion, the term "mail- 
box" is used to mean only the subscriber visible entity, 55 
and, where necessary, the term "message endpoint" is 
used to denote the implementation entity or entities 
which underlie the integrated mailbox. 



Integrated mailboxes have certain desirable and 
preferable characteristics. A fully integrated mailbox, in 
accordance with the present invention, includes the fol- 
lowing major capabilities that are not present in a single- 
media mailbox: 

a. The ability to deal with messages of different da- 
ta/information types, or having multiple parts (mul- 
timedia mailbox). 

b. A single inventory (message list), listing all mes- 
sages of all data types, with the ability to control 
presentation of the inventory (e.g., sort the invento- 
ry according to message type, priority, or time of de- 
posit, regardless of the type of message), with con- 
ceptually similar user interface actions for equiva- 
lent operations on any message type, and with the 
ability to randomly select messages for retrieval. 

c. Notification mechanism(s) which can be used to 
alert the user of the deposit of any type of message. 

d. The ability lo access the mailbox through a vari- 
ety of commonly-available mailbox access termi- 
nals (PC, DTMF phone, etc.), without special equip- 
ment, and with, as far as practicable, logically the 
same capabilities for all terminal types 

e. The ability to perform data typo conversions au- 
tomatically, in support of transparent multi-terminal 
user access, or upon subscriber explicit request 

f. The ability to receive and send messages to sub- 
scribers of. existing messaging systems, using a va- 
riety of widely-implemented messaging protocols. 

Note that there are degrees of integration in today's 
single-media mailboxes, both with respect of allowed 
message types and the access terminal types which can 
be used. For instance, integrated facsimile/voice mail- 
boxes are common today, and e-mail can be used to 
transfer non-text information. Similarly, e-mail mailbox- 
es cannot be accessed usingtelephones, and voice/fac- 
simile mailboxes cannot be accessed using a PC. 

Although it is possible to have a mailbox which is 
integrated with respect to multiple message types but 
which can only be accessed through a single type of ter- 
minal (e.g., e-mail systems using MIME), a fully integrat- 
ed mailbox is preferably accessible from several types 
of terminals and pathways, to maximize the subscriber's 
ability to access his messages in various circumstanc- 
es. The following terminal types are provided by the 
present invention: a. Conventional DTMF telephone 
handset; and b. Personal. Computer (PC). 

However, otherterminal types such as personal dig- 
ital assistants, cellular telephones, two way pagers, etc, 
can be used. - : 

It should bo emphasized that an integrated mailbox 
subscriber using the present invention is able to dynam- 
ically change the terminal used, from session to session. 

The integrated messaging system (IMS) of the 
present invention is preferably interfaced to external 
systems. This allows the subscriber to exchange mes- 
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sages wich external subscribers and can be used to in- 
tegrate several existing messaging accounts on differ- 
ent systems so that the user accesses a single (virtual) 
integrated mailbox. The following types of external sys- 
tems can be included: 

a. The Internet. 

b. Commercial subscription mail systems (usually 
X.400). 

c. Private mail systems (e.g., MS Mail, cc:Mail). 

d. CFE voicemail systems and other foreign net- 
work-based voicemail systems (e.g., the subscrib- 
er's cellular phone voice mailbox). 

There are several ways, in accordance with the 
present invention, in which integrated messaging can 
be realized. However, the preferred approaches are dis- 
cussed below. 

The integration of the mailbox can be real or virtual. 
"Real" mailbox integration means that the messages of 
all types are located in a single messaging system (MS), 
and that subscriber and administrative control facilities 
for messages and mailbox configuration parameters are 
provided at a single user interface point and do not in- 
volve cooperation or interaction with any other MS. "Vir- 
tual" integrated mailboxes provide the same subscriber- 
visible functionality, and appear the same to the sub- 
scriber as a real integrated mailbox. However, in the vir- 
tual integrated mailbox, the subscriber's messages are 
stored in at least two different MSs, whose configuration 
can be (but need not be) performed separately. The dif- 
ferent messaging systems cooperate to provide the 
complete functionality. The term "associated MS" is 
used to denote and MS that is in a special relationship 
with another MS for the purposes of synthesizing a vir- 
tual IMS, and the term "external MS" is used to denote 
an MS which is not so closely associated, but whith still 
has an interface to the IMS. 

The distinction between real and virtual integrated 
mailboxes is invisible to the subscriber. Real messaging 
systems can comprise multiple subsystems, such as the 
preferred distributed system described herein, with the 
"mailbox" spread across several pieces of hardware. 
Both types of integration need interfaced to external 
MSs, even if they are not part of a virtual IMS. The re- 
lationship between the MSs that are being integrated in- 
to a virtual IMS ("associated MSs") is much closer than 
that between MSs that just happen to interwork ("exter- 
nal MSs"). While an integrated mailbox system could be 
totally self-contained (allowing messaging only between 
its subscribers, like most voicemail systems today), it is 
preferable to be able to send and receive mail from other 
systems. Real mailbox integration is preferred and de- 
scribed in detail herein. 

Several approaches are available to create a virtual 
IMS and are described here: a: desktop, integration, b. 
front-end director, and c. pass-through integration.-- 

Desktop integration (DTI) is the emulation, as per- 



ceived by the subscriber, of an integrated mailbox, when 
a personal computer 60 (PC), as illustrated in figure 1. 
is used for message endpoint access, there are several 
non-integrated messaging systems (MS) such as e-mail 

5 system (EMS) 66 and voicemail (VMS) 68, and the in- 
tegration of the mailbox may not occur for non-PC ac- 
cess terminals; in other words, special-purpose PC soft- 
ware "does" the integration of the mailboxes; the MSs 
need no ability to handle non-native data or communi- 

'0 cate with each other. One approach to this integration 
is to use a conventional browser'with a local home page 
that includes link to the various messaging systems. An- 
other simple DTI approach is to use separate TCP/IP 
connections 62 and 64 to each MS 66 and 68 on a single 

'5 dialup point-to-point protocol (PPP) connection to a 
router 70 providing hardware dedicated to routing IP 
packets over various physical h'ardware interfaces, as 
shown in figure 1 . I 

DTI of this type is useful when no close integration 

20 of the MSs is possible lor political or administrative rea- 
sons. 

In another approach to virtual mailbox integration, 
a front-end director 80, as illustrated in figure 2 directly 
interfaces with the customer, thus avoiding changes in 

25 any of the integrated MSs 66 and 68. The director 80 
communicates on the back end with the separate MSs 
66 and 68 that need to be integrated. Two major variants 
of this approach are possible: the director 80 simply 
passes requests through in reai-time, and thus stores 

30 no messages itself; or, the MSs 66 and 6B forward mes- 
sages to the director 80 when they are deposited, and 
the director 80 stores them until the subscriber logs in. 
In the latter case, the director 80 effectively becomes a 
full IMS with external MS interfaces. 

35 • The director 80, if it does not store messages, ac- 
cepts subscriber requests, interprets them, and then 
communicates with the various rvlSs as needed to either 
retrieve or send messages and inventories. The director 
• 80 in this embodiment is essentially a transport-level or 

40 application-level relay, with some firewall-like function- 
ality for security. The back end network is typically a lo- 
cal area network (LAN) or high speed wide area network 
(WAN), onto which subscriber requests are multiplexed . 
However, to provide DTMF access, the front-end di- 

45 rector 80 has the necessary voice, facsimile, data and 
voice/data telephone interfaces of a system such as that 
described in U.S. Patent 5,193, 110. In practice the voice 
' user interface is a superset of the interface provided by 
the VMS 68 alone. ' 

so Pass-through integration is another approach 

where the front-end director 80 functionality is tightly in- 
corporated into one of the MSs, such as VMS 68. The 
MS deals with the messages of its native type, but acts 
as a real-time proxy for subscriber requests for other 

ss messages types. As discussed previously, the need for 
both voice and facsimile as wellias PC access, the dif- 
ficulty of interfacing in real-time to an external VMS, the 
• difficulty of augmenting an EMS to handle voice, and the 



DOCID: <EP 0845B94A2_I_> 



EP 0 845 894 A2 



8 



need for VMS-like notification mechanisms results in a 
preference that the director be added to a VMS 68 rather 
than to an EMS 66 as illustrated in figure 3. 

The foregoing discussion indicates that the prefer- 
able approaches to an integrated multimedia mailbox 
with both DTMF and PC access^are eithera full, real IM 
system, or enhancement of a VMS so that it can provide 
pass-though real-time access to other messaging sys- 
tems. As a result, there are two preferred system-level 
architectures: a. An enhanced VjMS (i.e., the IMS) which 
provides all message storage and all user interlaces for 
all types of message. All other' MSs interface to the IMS 
as external, non-integrated systems, b. An IMS which 
provides permanent storage for voice, video, text, e-mail 
and facsimile messages and exchanges other message 
types on demand with one or more closely associated 
systems such as an associated-EMS (in addition to any 
interfaces to other external MSs). The IMS as all user 
interfaces and passes through user commamds related 
to the associated EMS(s). The (MS exchanges deposit 
notifications with the associated EMS(s). 

Figure '4 illustrates the system/network architecture 
for either the real or virtual integrated mailbox and inte- 
grated message system (IMS).' As illustrated, the sys- 
tem architecture 100 allows access by a telephone 102 
through a public switched telephone network (PSTN) 
104 to the integrated messaging system 106 either di- 
rectly or through a modem 108^. Access by a personal 
computer 109 can also be accomplished through the 
PSTN 1 04. Access by a personal computer 1 1 0 through 
the Internet 112 using a modem 114 is also provided. 
Note the associated e-mail system (EMS) 115 is shown 
to depict the architecture of a' virtual system. The IMS 
106 is also coupled to other systems 116, 119 and 120. 
The content of the IMS 106 wiil be discussed in detail 
with respect to figure 6. j 

The system 1 00 can also be provided with a multiple 
integrations unit (MIU) subsystem (not shown) such as 
described in U.S. Patent 5,260,990. The MIU can host 
small numbers of PC sessions. This is, however, not a 
large-scale architecture. 

The present invention has the ability to receive, 
send, and store messages of several data types includ- 
ing voice, facsimile, multipart (i'e., voice-annotated fac- 
simile), video, text and e-mail messages. The platform 
1 32 (see figure 6) is designed to, accommodate the mas- 
sive amounts of storage required for video data. 

All messages have certairji information (the mes- 
sage envelope) that goes alc|ng with them, such as 
sender, date/time of deposit, length, etc. The informa- 
tion varies with message type and, to some degree, with 
the means by which the message was received. The en- 
velope information is preferablyjstorcd with the messag- 
es carried along with the message if it is to be delivered 
to an external system, and be presented to the subscrib- 
er. The IMS 106, as previously discussed, is able to 
present a single list or inventory, containing all messag- 
es of all types (sorted into types), to the subscriber when 



he logs into his mailbox, and provide the ability to select 
messages for retrieval. In addition, some of the mes- 
sage envelope information can be presented in the in- 
ventory. The amount of information presented in the in- 
5 ventory, and the format of presentation are determined 
largely by the human aspects of the access terminal; 
when the voice interface is used, the presentation is 
preferably limited to simple spoken message counts 
("You have three new voice messages, one new facsim- 

io ile message, and two new E-mail messages and one 
new video message. One voice message is urgent."), 
otherwise the subscriber may quickly get confused. For 
the same reason, complex inventory sorting, message 
selection or folder capabilities are preferably not provid- 

15 ed through the voice interfate, even though they can be 
if desired. However, a PC interface preferably shows 
much more information to the user without overloading 
the subscriber, and allows sophisticated operations 
. such as organizing messages into folders. 

20 Generation of integrated message inventories oc- 

curs naturally on a real IMS. A virtual IMS needs to get 
the individual inventories from the external systems 
. specified for the subscriber. This is considered below in 
the discussion of interfaces to other MSs. 

2S . The voice interface of tho present invention 
presents a spoken message inventory which gives 
counts of messages per type, and additional salient in- 
formation such as whether any are urgent. There are 
essentially two folders: new and saved messages, plus 

30 a virtual "wastebasket" which may be emptied (or not) 
at the end of the session. Selection of a message to piay 
is predetermined by the system (play only voice and 
text-to-speech type messages), with some limited ad- 
ministrator controls (e.g., play new or saved messages 

35 first) ; and the user cannot chose to select a specific 
message, other than by skipping forwards or backwards 
through the messages. • 

A PC interface according to the present invention 
provides an inventory much like the message list of e- 

.40 mail systems. Typically, it includes, for each message: 
type of message (voice, video, e-mail, facsimile), sub- 
ject (if any), sender, time of deposit, size of message 
(bytes, pages, seconds, as appropriate), and status 
(new/read, urgent, replied to. forwarded, etc.). 

45 in a PC interface the user clicks an inventory entry 
to select the message to be retrieved; the system re- 
trieves the appropriate message and provides it to the 
user in the proper format. The PC can support folders, 
so the subscriber can organize his messages. This can 

50 be done locally at the desktop, by moving messages to 
local file system directories or files. However, forthe pre- 
ferred type of PC interface (e.g., using a standard web 
browser), the folders can be implemented by the IMS 
106. 

ss Message headers preferably include important de- 

tails from the message envelope for non-voice/facsimile 
messages. The envelope of messages received via e- 
mail can have a lot more detail than these of voice/fac- 
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simile messages, and are also in text format. These type 
envelope elements are parsed by the system 106, and " 
spoken in the same way that sender mailbox number, 
date/lime, urgency and other envelope elements are 
handled (i.e., by concatenating pre-recorded prompt 5 
fragments). Preferably, the envelope data is stored in 
the same way, regardless of message type, to output 
the non-voice/facsimile envelope in spoken format. 

One problem area is in identifying the message 
sender to the subscriber. Ideally, the system should to 
speak the sender's name, preferably in the sender's own 
voice. For messages received from a sender on the 
same system, the system can access the sender's name 
announcement (or speak the mailbox number if there is 
no name announcement). The same is true of messag- T 5 
es from other systems connected through digital net- 
working using the Digital Messaging Network Version II 
(DMN II)™ system available from Boston Technology 
Inc. of Massachusetts. Digital Messaging Network Ver- 
sion II (DMN II)™ conforms lo the AMIS-D voice mes- 
saging protocol, which passes the name announcement 
of the sender along with the message, using X.400 and 
a real time DMN II protocol to retrieve remote name an- 
nouncements during message addressing between 
systems. 25 

For non-voice messages received from other ven- 
dor's platforms (including X.400 and MIME/SMTP mes- 
sages from the Internet), the sender address informa- 
tion is a character string, and there is no explicitly iden- 
tified name announcement. Of course, an X.400 or 30 
MIME voice body part can be included with any mes- 
sage, and so the sending system can send a name an- 
nouncement. However, the receiving system will not 
know it is the name announcement and just play it as 
part of the message body. An example of such an ad- 3S 
dress is "bill@abc.com". The receiving system 106 
parses this string and speaks in the same way as a mail- 
box number, character by character. Text-to-speech 
could also be used, but there are problems for today's 
text-to-speech technology in handling an infinite variety -to 
of strange words (such as "abc"). In any case, it is often 
clearer to spell out the address when spoken. A pre- 
ferred solution is to use the prompt voice fragment ap- 
proach with additional prompt fragments for common 
address elements, to output (for example) the voice 
prompts "b" "i" "I" "I" "at" "a" "b" "c" "dot" "com" for the • 
address noted above. This does not require text-to- 
speech capability. 

The need to handle messages from the external 
MSs 1 1 6-1 20 means that some voice messages will be so 
received in encodings other than the native format of the 
platform. Examples would be MIME messages contain- 
ing audio/wav body parts, or WAV files. The present in- 
vention can output this data to the telephone line just as 
easily as the native format using the freeware software ss 
package SOX. An alternative is to convert everything to 
the native format, but this can cause quality loss if-the 
data subsequently needs to be converted to some other'-'" 



format for output over same other interface and is not 
preferred. 

For output to a conventional telephone, non-voice 
messages of the text data type are converted to voice, 
or otherwise represented by vojee. However, there are 
some structures (such as the MIME multipart/parallell 
that may be too complicated, or riot useful, or impractical 
to output to a voice telephone, in such a situation, the 
system should indicate to the subscriber the type of the 
message and other envelope information, and explain 
that it cannot be retrieved from a DTMF telephone using 
a standard voice message. 

However, a picture-telephone can receive image 
type mail messages. 

Recording of a voice, facsimile or voice-annotated 
facsimile message over the voice interface is performed 
in a conventional manner. To generate other data types 
DTMF key entry is used to select from a menu of simple 
parameterized pre-canned messages. For instance, a 
caller can send text messages containing a callback 
phone number to a subscriber's cellular handset or pag- 
er. The present invention provides these capabilities for 
video, text, e-mail messages, as well as for other data 
types (facsimile back services are an example of similar 
usage). i 

The limitations of the DTMF keypad do restrict the 
ability to address the message to non-numeric address- 
es. In the present invention, however, a group address 
containing nonrnumeric addresses (defined using the 
PC interface) can be specified through the DTMF inter- 
face. The present invention also allows a reply with a 
voice message to any received message from any send- 
ing address. The system 106 records the voice and de- 
posits it in the local sender's mailbox, or converts it to 
an appropriate MIME, AMIS-D or X.400 body part using 

DMN II or the functions of a client e-mail reader such as 

i 

MSExchange, cc:Mail or Netscape Internet Mail, and 
sends it to the original message sender's address, if ex- 
ternal. 

Speech-to-text, or voice recognition, is also a 
means to send text messages from a conventional tel- 
ephone and also a way of addressing non-numeric ad- 
dresses (by spelling them out), j 

Forwarding is the copying of!a message and the de- 
posit of the copy in a different mailbox, or transmission 
to a third party. Forwarding of rion-voice messages is 
provided by the present invention through both the voice 
interface and the PC interface because when the mes- 
sage does not need to be converted to voice, such as 
for facsimile messages, the message can be sent either 
to another mailbox, or to a subscriber-entered facsimile 
telephone number, without actually retrieving the mes- 
sage. This approach is also used by the present inven- 
tion for other data types (video), as is the turnaround of 
lies for alternate facsimile/voice use. When the destina- 
tion is not the same IMS the message is format convert- 
ed as needed.. Forwarding to another mailbox on the 
same IMS is implemented in the same way as for voice 
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and facsimile, and operates independently of data type. 
Forwarding to a telephone number, when done automat- 
ically by the system, is considered a form of message 
deposit notification. The same mechanisms are used for 
forwarding, so we address it below. . . 

Facsimile machines may also be used to access the 
system 1 06 via the dialup voice interface; often, the fac- 
simile machine may actually be a PC running facsimile 
software. The system of the present invention has fac- 
simile modem capabilities, andhandles facsimile mes- 
sages in a variety of ways, often in conjunction with a 
DTMF telephone for control and listening to voice anno- 
tations. The system stores facsimile messages in their 
native form (tiff) and no conversion is necessary. The 
additional complexities for the |MS relate to the output 
of non-facsimile messages to facsimile machines. To do 
this the system converts text messages and non -facsim- 
ile image data to facsimile format using a function avail- 
able from Natural Microsystems or the public domain 
sollware pbmpius can be used; to convert to/from vari- 
ous picture forms. Exchange of facsimile data with ex- 
ternal systems is performed using both MIME and X.400 
for G3 facsimile body parts. 

For PC access, two physical interface types are pro- 
vided: dialup to the IMS telephone ports, and via the In- 
ternet (or another TCP/IP network). In addition, several 
ways to handle voice messages are provided: purely 
digital, where the voice data is simply transferred like 
any other type of data (such as by using a browser as 
previously discussed), and the PC turns it into audio; the 
use of a voice/data line-sharing scheme, such as pro- 
vided with the VoiceView system available from Radish 
(the latter would only be available through the IMS dial- 
up ports) and transfer of the'vlbice data via an e-mail 
attachment with the conversion^ audio occurring in the 
PC . 

Accommodating access from a PC (see figure 6) 
can be provided through an APU 150 and IPU 146 (NIU) 
hosting SMTP/MIME and POP protocols. Because of 
the rapid growth in Internet connectivity and on-line 
services, message user agents (MUAs) (or client mail 
readers) using these protocols are becoming universal 
for subscribers using dialup access. However, the Hy- 
pertext Transfer Protocol (HTTP) is the preferred proto- 
col since it is the one used to, interface with standard 
web browsers. It is a simple request/response protocol, 
which uses TCP/IP. Requests (called methods) are pro- 
vided lo gel and create objects; (real or synthesized da- 
ta), and to do other operations in support of navigating 
a global, interconnected set of information. The subjects 
of the methods are identified by a Universal Resource 
Identifier (URI) or Locator (URL) which specifies the lo- 
cation (including the Internet name of the host whore the 
information is stored) and the rpeans to access the ob- 
ject. Responses are returned to a requester in MIME- 
compatible format, so the MIME content-type and con- 
tent-encoding can be determined by the requester, and 
the object presented in the appropriate way. To actually 



provide the "web" of connected objects, specially -for- 
matted text scripts or templates, using the Hypertext 
Markup Language (HTML), are used. These "web pag- 
es" can have embedded links to other objects systems/ 
s hosts), as well as presentation control capabilities. The 
browser follows these links by using the linked object's 
URI to send a GET request to the host identified in the 
URI, when the user selects the URI. By putting links to 
further pages (HTML static documents, or dynamically 
io synthesized documents) in a web page, a hierarchical 
organization of information can be established. HTML 
. and HTTP also support entry of data via forms, access 
to files via FTP, and interfaces to other information sys- 
tems, such as e-mail, GOPHER, and News. 
■ .is To provide the PC 1 09 or 1 1 0 (see figure 4) with web 
browser^ access, the IMS 106 of the present invention 
provides an HTTP server (IPU 146) to handle the re- 
quests from the browser, and provides an organization 
of the information in the IMS 106 into a logical structure, 
20 using HTML pages as will be discussed in more detail 
later. - 

In general, HTTP servers are widely available, both 
as public domain source code, and as commercial prod- 
. ucts. Toolkits are available to simplify page creation. HT- 
2S TP sorvors are highly configurable, including the map-" 
ping of universal resource locators (URLs) to internal (or 
external) data or operations (executable scripts). 

A simplified example of an HTML page hierarchy 
for the messages stored in the IMS and the subscriber 
30 operations used to access messages is given below. 
Sample URLs are given to aid understanding. 

a. Subscriber (or caller) uses a browser 144 (see 
figure 6) on a PC 142 to open subscriber's IMS 
35 home page (e.g., "http://www.mail. somerboc. com./ 
JoeQUser/", or "http://www.mail.somerboc.com/ 
61 7-246-9000/") or http://www.mail.somerboc.com/ 
awscripts/btv.dll?REFRESH. A "bookmark" could 
also be used to remember the URL. 
40 b. The PC 142 dials directly to the IMS 132 or via 
an Internet service provider (ISP, 140, and connects 
to the IMS host 132. The PC 142 sends an HTTP 
GET method for the home page URI. 
c. The I MS HTTP server 1 40 sends the home page 
4S in the response to the GET, and the PC's browser 
144 displays it. The home page has welcome text 
and/or graphics and/or voice announcement, in- 
cluding a password entry field. For users wishing to 
leave a message for the subscriber instead of log- 
so ging in, there is preferably a "button" on the sub- 
scriber's home page to "leave a message." The call- 
ing line identifier (CLlj can be used to verify that the 
calling number matches the subscriber name, or for 
using the authentication capabilities of HTTP. A se- 
S5 cure socket layer (SSL) is used to provide an initial 
secure connection for authentication. A menu of ad- 
ditional services besides integrated messaging can 
also be provided. 
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d. Using the browser 144, the subscriber enters the 
password in the login information form, the browser 
144 sends it to the HTTP server 146. The server 
146 validates the password and synthesizes the 
subscriber's main messaging page ("http://www. s 
mail.somerboc.com/JoeQUser/ inbox") from the 
contents of the subscriber's message store, and re- 
turns the page to the browser 1 44. The subscriber's 
messaging page contains links to each' of his stored ; 
messages ("http://www.mail. sorrierboc.com/aw- io 
scripts/btv.dll?REFRESH) and whatever inventory 
information ("http://www.mail.somerboc.com/aw- 
scripts/btv.dll?DRTR&Unique Msgld) is desired for 
display, plus buttons for sending, deleting, forward- . 
ing or other message actions. -.is 

e. If the user clicks a message link, the message is 
retrieved by the HTTP server 145 from wherever it 
is stored. The message is formatted and encoded 
as a MIME message, and is presented by the 
browser 144 according to the MIME type (lext, irrv 20 
age, video, voice, etc.). Voice and facsimile mes- 
sages stored need to be converted to MIME format 
first. 

f. Buttons are also links to URIs. If the subscriber 
clicks a button, the HTTP server 146 may simply 2S 
use the URI as a command name and perform the 
action, or it may return an HTML form to allow fur- 
ther user input, or to give more details about some 
entity. 

.30 

In the present invention, an HTTP server is provided 
in each application processing unit (APU) 150 (see fig- 
ure 6) that is configured to handle data calls. In addition, 

. an HTTP server (IPU 146) is provided for subscribers, 
accessing the IMS 132 via an ISP 146 and the Internet 35 

■ 136. For small numbers of concurrent PC sessions, the 
MIU previously mentioned, with modem banks, can also 
be used to provide PC dial-up access. Note that all HT- 
TP servers are configured identically, and use the dis- 
tributed subscriber database access mechanisms (in 40 
the same way as a voice application) to locate and re- 
trieve subscriber data and messages, even when they 
are not on the same APU. The HTTP servers use the 
existing platform operating system TCP/IP and PPP 
protocol capabilities. 4s 

Two situations for data type conversion can arise: 
when a subscriber's terminal type will not accept the 
stored message formal, and upon user request An in- 
termediate situation is when a subscriber requests de- 
livery or forwarding of the message to a system or ter- so 
minal, other than the one he is using, which does not 
support the data type. Most conversions are implicit 
from the message typo and the destination, but it may 
be preferable for a subscriber to explicitly request con- 
version (e.g.) of a facsimile message to text for forward- ss 
ing to an Internet address, even though the message ; 
could have been sent as a MIME facsimile message. ■ 
For the IMS of the present invention, messages are 
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preferably stored either in native voice format as de- 
scribed in U.S. Patents 5,029,199 and 5,193,110, native 
facsimile lormat (preferably conventional tiff), or in a 
MIME format. For PC access, or delivery to a PC via 
outdial, handling of dala other than native voice data is 
preferably by sending it unconverted using the stored 
MIME format. Facsimile information is preferably sent 
in the MIME image/tiff format. Voice data is preferably 
sent as the MIME audio/wav type. 

Notifications are used primarily to alert a subscriber 
that he has messages, so the subscriber need only ac- 
cess the system when messages exist. Notification 
mechanisms vary from the communication of one bit of 
information (messages present or not), to delivery of the 
actual messages themselves. ; 

The present invention, using Access NP' M , pro- 
vides comprehensive notification delivery capabilities 
including: paging via outdial, TAP and TNPP: message 
waiting indication (MWI), via SMDI, and using various 
SS7 or ISDN capabilities; special delivery outdial (deliv- 
ery of voice and facsimile messages by outdialing the 
subscriber at a specified telephone number); and cellu- 
lar short messages, containing mailbox message 
counts or callback numbers. ', 

Most of the above do not actually deliver messages, 
and so require no message data type conversions. Spe- 
cial delivery, however, is more complicated. The desti- 
nation terminal must be capable of receiving the data 
type, or the system must be capable of recognizing the 
terminal type and converting the message accordingly. 
The data conversions discussed (e.g., text-to-facsimile 
or text-to-speech) should, however, handle most notifi- 
cations. Special delivery via outdial to a PC is also pro- 
vided. ■ ; 

In addition to data conversion for special delivery, it 
is necessary to consider how other message types af- 
fect the notification algorithms. One approach is to han- 
dle them in exactly the same way, so that any e-mail 
message causes the MWI light of the subscribers tele- 
phone to come on, or causes a page to a pager if it is 
marked urgent. 

A real IMS is self-containedifor notifications, in that 
' any message which is to be notified is stored in the IMS. 
However, in a virtual IMS, one MS does not store all 
messages. Further, some of the integrated message 
systems may have no notification capability. Thus, there 
needs to be a way to communicate the presence of mes- 
sages between MSs ("cross-nolificalion"). This may re- 
quire non-standard mechanisms, since most MSs are 
not designed to accept notifications. This, in turn, may 
require MS modifications, which defeats much of the 
purpose of a virtual IMS. Below iwe discuss how to pro- 
vide VMS exchange notifications with an associated 
EMS, without major EMS modifications. 

An associated e-mail system (EMS) will preferably 
have one of SMTP/MIME and POP protocol capabilities 
(or equivalent X.400 capabilities). The EMS can initiate 
SMTP sessions, but the VMS must initiate POP ses- 
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sions. The EMS is configured to automatically create a 
copy of each message deposited, or an additional mes- 
sage with inventory information, give the message a 
special VMS recipient address, and send it via SMTP to 
the VMS (IMS 106). This has desired asynchronous no- £ 
tification characteristics. The VMS (IMS 106) receives 
the message, parses it (or merely notices it), and uses 
the information to control the notification mechanisms 
(but not score it). Alternatively and less preferably, the 
VMS (IMS 106) uses POP to poll the subscriber's mes- io 
sage inventory. This is a periodjic event, likely to cause 
a lot of traffic, since it needs to be fairly frequent and 
there could be large numbers of subscribers. In either 
situation, the VMS (IMS 1 06) needs to be provided with 
the subscriber's e-mail address' and password. is 

There is also a hybrid of the above-discussed ap- 
proaches where the VMS is ah IMS, and it uses POP to 
retrieve EMS messages but alsb offers POP to the sub- 
scriber so that client e-mail SW connects to the VMS 
(IMS) to retrieve e-mail messages. * 20 

For notifications from the VMS to the EMS, the VMS 
sends an e-mail text message containing the voice/fac- . 
simile message inventory using SMTP. 

For both real and virtual IMSs, there is a need to 
interface- with external systems' 116-120 using the reg- 25 
ular store-and-forward e-mail paradigm, in order to 
make the IMS subscribers part of a single world-wide 
messaging community. For the virtual IMS, special con- 
sideration must be given to the "associated MSs", since 
that non-standard (for e-mail syjstems) but conventional 30 
techniques need to be applied.! 

The Access (MP™ platform available from Boston 
Technology, Inc. provides AMIS-D digital) and AMIS-A 
(analog) networking for transferring messages to other 
VMSs with the requisite protocol support. This includes 35 
most VMS vendors. The system handles voice data ac- 
cording to the AMIS-D specification. It also has down- 
ward-compatible enhancements for interworking with 
other systems, so that multiple ; Boston Technology IMS 
clusters become essentially connected in near real- 40 
time. 

For interfacing with externa'l VMSs, or between Bol- 
ton Technology, Inc. IMS platforms, AMIS-D is pre- 
ferred. The X400 technology js also usable for inter- 
working with other X.400 mailjsystems, allowing con- 45 
nectivity through X.400 administrations to many e-mail 
users. Since almost all X.400 public mail systems have 
Internet connectivity, this is a preferred approach to pro- 
viding world-wide connectivity for IMS subscribers. 

When external MSs do not have X.400 interfaces, so 
MIME/SMTP protocols are the next most preferred 
choice, both for Internet connectivity and for connection 
to corporate LAN mail gateways. 

For a virtual integrated messaging system (IMS), 
some of the systems outside of the VMS 68 need special ss 
treatment, so that the message, endpoints of the MS (for 
example, e-mail system 66) appear to be part of the 
same integrated mailbox as the VMS message end- 



point. Interfaces and operation for the "pass-through di- 
rector" approach to virtual IMSs will be discussed in 
more detail (notifications have been addressed above 
already). As for notifications, an EMS with SMTP and 
POP capabilities, connected as shown in figure 3, is as- 
sumed. 

The general handling of a virtual IMS subscriber's 
operations in a virtual IMS will be described below for 
each of the typical phases of a session. 

When the subscriber logs into the VMS 68, the VMS 
68 generates an integrated message inventory. This is 
done through the use of a POP session between the 
VMS 68 and the EMS(s) 66. The VMS 68 logs into the 
POP server on behalf of the subscriber (POP USER and 
PASS commands), queries the EMS inventory (POP 
LIST command) and merges it with the local inventory. 
The VMS 68 maintains a map from the POP message 
IDs to an identification (place in sequence, ID number 
or synthesized URI) used between the VMS 66 and the 
subscriber. Locally, stored messages are likewise given 
identifiers. 

To select a message for retrieval, the subscriber in- 
terface may use POP (for MUA interfaces), an HTTP 
GET method (for web browser interface access), or en- 
tor DTMF commands (for voice access). In any event, 
the VMS 68 maps the subscriber-requested message 
ID or URI to the EMS message ID, or to an internal mes- 
sage ID. If the message is on the EMS 66, the EMS 66 
uses POP to retrieve the message from the VMS 68 
(POP RETR <msg id> command), and outputs it to the 
subscriber. If the message resides on the VMS 68, it is 
simply retrieved and played. Data conversions, if nec- 
essary, are performed as described earlier. 

To submit a message, the VMS 68 needs to deter- 
mine whether the message is to be sent to the EMS 66 
or handled by the VMS 68. If the former, then the DMN 
II system previously mentioned (using the X.400 proto- 
col) is used to forward the message to the EMS 66. The 
criteria for the decision would typically include the recip- 
ient address and the message data type. There are 
many possibilities for the algorithm; the preferred is: 

a. If the message is native voice or facsimile (i.e., 
using the DTMF interface), and the recipient ad- 
dress is a phone number, the message is handled 
in the conventional VMS manner for a voice or fac- 
simile message. 

b. II the message is native voice or facsimile, and 
the recipient address is not a phone number, the 
message is sent to the EMS 66, with the data con- 
verted to a MIME audio or image/tiff type. 

c. If the message is a MIME audio or facsimile type 
(i.e., SMTP or HTTP user interface), and the recip- 
ient address is a phone number, the data is convert- 
ed to native format and handled as conventionally 
by the VMS (note: addressing to a phone number 
could be prohibited to avoid this conversion, initial- 
ly) 
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d. If the message is any MIME type, and the recip- 
ient address is not a phone number, the message 
is sent unchanged to the EMS 66. 

e. When messages are sent to the EMS 66 for han- 
dling, the sender address is set to the e-mail user 
name of the subscriber, so it appears that it origi- 
nated in the EMS 66. 

The approach discussed above is a straight forward 
way of operating. There are much more complicated 
ways of doing it. The advantage of the simple approach 
is that the VMS involvement in handling e-mail messag- 
es is minimized. 

Figure 5 illustrates a general functional architecture 
for a real IMS which provides the capabilities discussed 
above. A more detailed description will be provided with 
respect to figure 1 . 

Various elements of the archictecture will be dis- 
cussed. Note that figure 5 is not internded to be a defin- 
itive and complete representation of Ihe actual software 
architecture of the IMS which will be described in more 
detail with respect to figure 6. 

The dialup interfaces 130 in the APUs 150 have 
drivers for access to conventional DSP hardware, pro- 
viding voico interface! functions (V), facsimile modem 
capabilities (F), data modem capabilities (D) and 
VoiceView voice/data modem capabilities (V/D). 

For DTMF access, the voice application 131 is the 
application described in the related patent previously 
mentioned, which accommodates multimedia messag- 
es and data conversions, for example text-to-speech 
and text-to-facsimile. 

The multimedia message store 132 is distributed 
across all the APUs 150, with most of a subscriber's 
messages being stored on a "home" APU. Standard 
methods are used by applications in any subsystem to 
access the distributed message store in a location-inde- 
pendent way. 

The Boston Technology, Inc. Digital Messaging Net- 
work Version II (DMN II)™ product 133 provides store- 
and-forward communication with other MSs, including 
other Boston Technology I MSs or VMSs, other vendor's 
VMSs, and external EMSs. Either the AMIS-D protocol 
1 34 or SMTP 1 35 is used for this. DMN 1 1 is automatically 
activated when a message requires delivery to an ex- 
ternal address. 

The subscriber PC access functions reside in the 
APU 1 50 for dialup access, and in the Nl U (I PU 1 46) for 
network access. Both instances of this set of functions 
are essentially the same, except that the datalink proto- 
cols are different. 

An HTTP server 136 is provided in the IPU 166 to 
allow access using conventional web browsers and as- 
sociated web pages 1 37, and SMTP and POP servers 
1 38 are provided to allow access from MU As. All other 
services POP3/IMAP4/Voice/PPP/X.400"are Utilized on 
other components and in turn utilize the IPU 166 for HT- 
TP if necessary. 



The PC application 1 40 provides the structure of the 
user interlace for HTTP access, and the necessary 
"glue" to interface with the internal system mechanisms 
(in this case, just the message storage). 

5 Microsoft WinSock 1 42 is used to provide the TCP/ 

IP protocol support, both externally and internally. Win- 
Sock is an integral part of Windows NT, and includes 
support for VoiceView data transfers. For Boston Tech- 
nology, Inc. platforms using the UnixWare OS, the TCP/ 

io IP support is also part of the OS, but VoiceView is not 
supported. 

The "real" IMS system 130 (106) of the present in- 
vention is preferably implemented using a distributed ar- 
chitecture such as illustrated in figure 6. The system 1 30 

15 includes a message platform 1 32 that is connected to 
both the public switched telephone network 134 (via a 
digital matrix switch 135) and the Internet 136. A sub- 
scriber can access the platform -132 using a telephone 
138 to perform message access functions such as re- 

20 trieving and listening to voice mail messages, lorward- 
ing messages, recording messages, and converting and 
playing facsimile messages. A detailed description of 
this type of access can be found in the U.S. patents pre- 
viously described and is available in the CO Access® 

25 and Access NP™ systems from Boston Tochnolohy, Inc. 
The subscriber can also access ihe platform 1 32 for ac- 
cessing voice mail messages and other types of mes- 
sages such as facsimile and video messages over the 
Internet 1 36 through a conventional Internet service pro- 

30 vider system (ISP) 140 using a personal computer 142. 
The personal computer 142 is a typical convention- 
al multimedia personal computer capable of running or 
executing an Internet browser 144, such as the pre- 
ferred Netscape 3.0 browser available from Netscape 

35 Communications or Internet Explorer 3.0 available from 
Microsoft, and capable of connecting to an I ntemet serv- 
ice provider 140. The PC 142 can also, of course, be 
connected to the Internet through a company local area 
network (LAN) via a high speed connection. The com- 

40 puter 142 preferably includes a modem with a speed of 
at least 14.4 Kbps and preferably at least 28.8 Kbps 
when the user intends to access video images. The 
computer 1 42 also includes a conventional sound card 
and associated audio speaker arid audio software such 

■*5 as the preferred TrueSpeech available from the DSP 
Group. For the recording of voice messages that will be 
transmitted to other mailboxes this computer 142 needs 
a microphone half duplex recording board such as 
SoundBlaster from Creative, Inc and software such as 

so MediaPlayer from Microsoft. For displaying still images, 
such as facsimile mail, the computer 142 needs 1o in- 
clude a tiff reader such as Microsoft Facsimile available 
from Microsoft Corp. These still image components can 
also be used to record facsimile messages fortransmis- 

55 sion to other mail boxes. For the playing of motion video 
images, the computer 142 uses ActiveMovie from Mi- 
crosoft. If video images for broadcast are to be recorded 
the computer 1 42 needs a conventional camera and as- 
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sociated software such as Connectix from Connectix, 
Inc. 

The browser 144 is used in a convenlional method 
to access an Internet web page hosted by the platform 
132. The browser 144 allows the user to perform func- 
tions such as viewing a message list, view addresses of 
voice/video mail recipients and senders as well as e- 
mail addresses, playing and saving messages, forward- 
ing messages to others, replying to messages, creating, 
listening to and modifying personal greetings, name an- 
nouncements and prompts, etc. jusing the graphical user 
interface and the video/sound fcapabilities of the com- 
puter 1 42. j 

The user accesses the platform 1 32 through one or 
more Internet processing units j(IPUs) 146 over a con- 
ventional digital communication path typically used for 
high speed access (10-100 Mb/s of a home page server. 
The unit 146 is preferably a pentium based computer 
that operates at 133/166 MHz with 32 MB of RAM and 
4GB hard disk drives that are mirrored, A convenlional 
Internet router (not shown), such as available from Cis- 
co Systems or Bay Networks which performs packet fil- 
tering, can also be provided between the Internet 1 36 
and the unit 146. The router can provide a "bridge" be- 
twocn the cthornet 149 and the network backbone. In 
such a configuration the router and unit 146 would be 
coupled preferably using a 100Mb Ethernet connection 
and perform the primary function of moving packets to 
and from the IPU 146. The unit t46 provides firewall pro- 
tection from hackers, acts as a web server for the mes- 
saging application, performs hypertext markup lan- 
guage (HTML) generation and; performs the voice en- 
coding and image encoding for the messages streamed 
to the computer 142. ; 

During a typical session a user will access the plat- 
form 132 over the Internet 136 using a standard web 
browser to obtain a service provider home page where 
the user will log into the Internet service provided by the 
platform 1 32. During this process the user is required to 
enter a mailbox identifier and 'a pass code which are 
checked to ensure that the user is authorized. Once au- 
thorization is confirmed a service session is initiated and 
the user is presented a page that includes a menu of 
service options such as viewing a message list, admin- 
istering mail box options, other network service fea- 
tures, etc. i 

When the users e-mail is stored and supported by 
another platform the message list can include a cross 
notification of the existence of an e-mail message in the 
mailbox message list. 

During a typical message retrieval function, the unit 
146 accesses a master control [unit 148 over an internal 
dual channel ethernet 149 to locate the storage location 
of the various types of messages stored for a subscriber 
and generates a web page which is transmitted to the 
personal computer 142 and which includes a list ol the 
messages. The user selects a message in a convention- 
al fashion by double clicking on a message descriptor 



or selecting the message and clicking on an appropriate 
icon such as "Play" in the display of the browser 144. 
The unit 146 responds by obtaining the selected mes- 
sage from the application processing unit 150 that 

s stores the message, converting the message into the 
proper format and transmitting it. In the case of voice 
messages the voice data is converted from the encoding 
for storage into a file in the encoding for playing used 
by a conventional audio application executable by the 

io browser 1 44 such as the preferred ActiveMovie f rom Mi- 
crosoft and streamed to the browser 1 44 where it is 
played as it is received. For facsimile and other text mes- 
sages a t iff file is created and transmitted to the browser 
144. For video messages the video data, if necessary, 

,s is conversed into the avi, mpg, mpeg, cu, etc. file formats 
• that allow the data to be streamed to the browser and 
displayed in a pop-up window in real-time as it arrives. 
That is, video as well as audio messages are played or 
displayed as received by the computer 142. During play 

20 back the user can perform the convenlional f unclions of 
rewinding, pausing, fast-forwarding, skipping, etc. The 
user can also perform operations associated with saving 
the message, deleting it or forwarding it to others. The 
processes performed by the Internet processing unit 

25 146 as well as those performed by the master control 
unit 148 and application processing units 50 and other 
units 52, will be discussed in more detail later and are 
set forth in the source code appendix which code can 
' be stored on/in various types of media, such as various 

30 types of disks and various type of computer memories, 
in the platform 132. , 

The operations or entry points available to a user 
interacting with the browser 144 include the following 
functions: 

35 

. QUIT- Exits out or logoff of the system and de- 

lete, all session information. 
DELE - Removes one or more messages from 

the system. 
40 SAVE - Saves a message. 

DRTR/RMSG - Retrieves data such as an audio mes- 
sage. 

RECORD - . Sets and starts record/send process. 

REFRESH- Gathers data and instantiates to brows- 
es er in HTML. 

USE- Selects a specific template. 

GRTR - Retrieves particular group list data. 

GDEL - Deletes a group list. 

GINS - Adds a single entry to group list. 

50 GUSE - Get group and use template. 

GPUT - Modify current group lists. 

: GNEW - Make new list. . 

MBOXADM - Administer user mailbox features such 
as change the password. 

55 

Each of these operations has a routine of a correspond- 
ing name in the source code appendix. The processes 
for the above-listed functions perform many of the same 
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operations such as authenticating a request, however, 
for simplicity of explanation three representative proc- 
esses, the process that refreshes (REFRESH) the 
browser display, the processes (DRTR/RMSG) that 
download messages for playing/displaying and the 
process records (RECORD) messages to send to oth- 
ers, will be discussed. The other processes are de- 
scribed in detail within the source code appendix. 

The refresh operation steps, with the data, ac- 
cessed, are depicted in figure 7. The refresh operation 
is integral to the display update operation of the inven- 
tion because each of the web pages that are transmitted 
to the computer 42 is affixed with an expiration date, 
such as yesterday, that causes the browser 144 to re- 
quest a reload of the page each time the page is ac- 
cessed or additional information is requested. This re- 
sults in the message list being updated to include mes- 
sages that have arrived since the current session start- 
ed. The refresh operation is also used to provide web 
pages that the user has selected during browsing where 
the refreshed page has never been transmitted. The re- 
fresh operation starts with a refresh request (a "GET" 
with a URI) being sent 1 to the Internet processing unit 
146. The IIS routine 170 denies access if the authenti- 
cation for the request is not present. If the authentication 
is present the request is passed 2 to the security filter 
routine 172. The security filter 172 sends 3 the request 
to the support 1 74 for authentication by the platform 1 32. 
The routine 174 checks 4 the session information data 

175 to see if a "session key" currently exists for the re- 
quest and if so the flow skips to step 9 discussed below. 
If the session key does not exist the authentication re- 
quest is sent to UNIX support routine 176 of the MCU 
148 and/or APU 150 for authentication. The authentica-. 
tion involves accessing 6 the user authentication data 
178 and returning 7 an indicator of success or failure at 
the authentication task The authentication successful 
result is returned 8 to the security filter 172 otherwise 
processing proceeds to step 21 where a message con- 
cerning the authentication failure is returned to and dis- 
played by the browser 144. The security filter 172 trans- 
fers 9 control to the validation routine 1 80 where account 
and file systems permission validation is performed and 
which returns 10 the result of validation. If validation is 
not successful, control transfers to step 21 where a fail- 
ure message is sent to the browser 1 44. If the validation 
is successful control transfers 1 1 and 1 2 through the Mi- 
crosoft Internet Information Server (IIS) routine 170 lo 
the template update routine 182. Routine 182 transfers 
1 3 the control to routine 174 to access 14 and check the 
state of the session cache 175. Once this check is com- 
pleted a request 15 for an update of the message/group 
information is presented to the routine 176. The routine 

176 obtains 16 the mailbox list information or group list 
information 177 stored in the MCU 148 and returns 17 
the information to the unit 146 where the session cache 
is updated. The updated data is forwarded 1 9 to4he 
HTML template update routine 182 where the current 



template from the template file 34 is instantiated 19 to 
match the data in the session cache 175. The data of 
the template is passed 20 to the IIS routine 170 where 
the HTML template for the page is returned 21 to the 

5 browser 144 which displays the refreshed page to the 
user. The templates preferably use the Microsoft. htx file 
syntax and the templates include standardized. varia- 
bles for the various data, such as <%AccountNum- 
ber%>, <%From%>, <%Media%>, <%Length%>, etc. 

io , The transmission of the HTML for a template allows 
the service provider to customize the home page (for 
example, by adding advertisements) and deliver pages 
• based oh the domain or class of jservice of the subscrib- 
er. 

'5 A refreshed active page that is designed to list 
stored messages can be formatted as illustrated in fig- 
ure 8. The messages can be iaccessed (played/dis- 
played in real time), saved, deleted from the platform 
database and in saved to storage local to the personal 

20 computer 1 42 using the conventional point an click par- 
adigm. 1 

The group list administration functions can be facil- 
itated using an active page such as depicted in figure 9 
where custom distributiin lists can be easily created and 

■25 edited to add, delete and modify addresses of recipients 
of the various type of messages that are supported such 
as voice, video, facsimile, telex, e-mail, etc with ad- 
dresses such as telephone numbers, e-mail addresses, 
universal resource location addresses, cable address- 

30 es, etc. 

The processing of a message retrieve or play re- 
quest (DRTR/RMSG), like the Refresh request, starts 
with the transmission of a request 1 by the browser 144 
to the unit 1 46 as depicted in figure 1 0. The request can 

as be made by double clicking on a message in the list of 
figure 8. The request is processed in the same way as 
the refresh request for steps 2-10. In step 11 the name 
of the requested file which has been made using a cod- 
ed name may also be converted to the real file name, 

40 which is a security issue which will be discussed in more 
detail later. The file request is pabsed 1 2 to the gateway 
routine 182. The routine 182 passes the request for the 
file to the support routine 1 74 which obtains 1 4 a unique 
identifier from the session information 1 75. The request 

■*5 is then provided 1 5 to the application processing unit 
150. The processing unit 150 accesses 16 the list of 
messages 177 and obtains 16. 5' the message data 186 
(voice, text, video, etc.) from storage. The message data 
passed to the support routine 1 74 and is converted from 

so the native storage format (from Oki 24 when the mes- 
sage is a voice message) into and stored 19.5 in cache 
or temporary storage 188. The cache storage allows lat- 
■ er requests for the same message to be processed with- 
out again decompressing the data. As the data is tem- 

55 porarily stored it is retrieved as needed to perform a real- 
time conversion into streaming mode data and into the 
desired format, such as TrueSpeech. This conversion 
will convert voice data into data compatible with Net- 
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scape type °plug-in° play back systems that Include 
TrueSpeech, Voxware, Real Audio and WAV (which is 
standard for Windows 95/NT). If the data is lext data it 
can be converted into a tiff file:compatible with a con- 
ventional tiff viewer. A facsimile message is typically 5 
stored in a tiff file. If the data is video data it is converted 
into MediaPlayer data (an .avi file) compatible with the 
MediaPlayer system available from Microsoft. The data 
can also be converted into JPEG or MPEG or still and 
motion video players available for conventional brows- io 
ers. The real-time streamed data, which includes the 
content type, is sent 18 to the IIS routine 170 and im- 
mediately forwarded 19 on to the browser 144 with an 
option to store the data in the cafche local to the unit 1 46. 
When streamed the browser 1 44 does not know the con- *S 
tent type of the message until the message is received 
from the server 146 and the invention relies on the de- 
fault mode of the plug-in that handles the data type. At 
the browser 144 the transmitted data, if it is Voxware 
data, causes a window to open and it is immediately zo 
played or displayed as the case may be. Other types of 
data such as RealAudio require that a play button in a 
pop-up window be activated. 

Once a message is received the subscriber, using 
the browser 144, can savo it locally, play it again, re- 25 
verse it, skip forward and backward, copy it into other 
files and other types of operations that can be performed 
with multimedia files. 

During a record process, a; media player, delivered 
with a conventional operating system such as windows 30 
95/NT, records the message or.the subscriber activates 
a plug-in of the browser 144 to record the message 
(voice, video, text, etc.). Using , the application, the sub- 
scriber edits, rerecords, etc. the message until satisfied. 
When finished the user provides a file name to the re- 35 
cording and stores it locally. The browser 144 is then 
used to POST the message to the server. Alternatively, 
the browser 144 can be used'to request that the server 
146 record the message andjsend it. The server re- 
sponds by performing the steps 1 -21 for refreshing a 40 
page previously discussed and forwards a template for 
a page, such as depicted in figure 11. The subscriber 
completes the template by providing the file name of the 
recording, addresses (such as telephone numbers, e- 
mail addresses, etc.) of the recipient along with indica- ts 
tors for privacy, etc. The browser 144 when the submit 
message" button is pressed creates a message header, , 
attaches the file and sends the file to the server 146. 
The server 146, when the file appears In the incoming 
message directory, converts the message into an ap- so 
propriate storage format (compresses it if necessary) 
and stores the message. The fjeader is reviewed to as- 
certain tho recipients addresses and the message is re- 
trieved and sent to the recipients. For example, a voice 
message to a particular telephone number would result ss 
in an outdial process being performed. If the recipient is 
a subscriber the message is copied to the recipients 
mailbox. ; 
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The present invention enhances the security of 
each session with a number of different features. 

Unit 126 is preferably limited to the HTTP (Hyper- 
text Transfer Protocol) and SSL (Secure Sockets Layer) 
type Internet service to help eliminate problems associ- 
ated with unsecure protocols. The present invention al- 
so has certain authentication features. A request is first 
sent from the browser 144 to the server 146 that does 
not include authentication. The Microsoft Internet Infor- 
mation Server (IIS-170) software running in the server 
146 checks this request. The filter 170 also checks to 
see if the user is valid and since there is no authentica- 
tion the check fails and the browser 144 sends the re- 
quest again this time with authentication. The request 
either passes or fails based on the authentication. When 
the request fails the browser 144 is notified. When the 
request passes the request is forwarded for processing. 
In an initial log-in situation the request is forwarded to 
the users home APU 150 where the validity of the user 
is checked again to determine whether the user is a sub- 
scriber. This double validation helps prevent nonsub- 
: scribers from obtaining access to the system. When the 
user is a subscriberthe user gets a message list sent to 
the browser 144 where it is displayed. Once a session 
is established the user it essentially communicating with 
the home APU 150 forthis subscriber which controls fur- 
ther transactions. All further requests by the browser 
144 to the server 146 go through the first level of au- 
thentication. 

The system preferably uses secure socket layer 
(SSL) packet encryption. 

The present invention also is preferably implement- 
ed using dual homed-host Internet processing units that 
prevents packet sniffing on the internal ethernet. A dual 
homed-host is a host that has two IP addresses that cor- 
respond to one or more physical addresses allowing it 
to be configured differently based on the IP address. For 
example, one IP could be configured only to work with 
SSL active and the other IP is used in the clear, i.e. with- 
out SSL. 

The provision of a router to perform packet filtering 
prevents source address spoofing. 

The present invention also assigns session num- 
bers and specially created tile names to tiles that are 
transferred to the browser 44. In this filter operation the 
process removes all correlation to any data set internal 
to the platform 1 32 from data sent over the network 1 36. 
For example, a message identifier that is sent to the 
browser includes a session identifier and a randomly as- 
signed file identifier (which can be the current time of 
day). The server 1 46 creates a session information entry 
that identifies the file. for the session identifier and the 
randomly assigned file identifier. When the browser 1 44 
requests the file the session identifier and previously as- 
signed file identifier are included with the request. The 
server 146 uses the session information to convert the 
file name into a real file name to retrieve the file. When 
the server 146 sends the requested file to the browser 
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144 the server 146. if it is not a streamed data tile, the 
server 146 assigns a pseudo file name that includes a 
session number and a randomly created file name. This 
pseudo file name and the real file name are also stored 
in the session information so that the file can be request- 
ed again. The session mumber is part of the create iden- 
tifier to allow communication that use the same random 
number to be distinguished. If the file is a streamed data 
file, the data of the file with content type is sent without 
any file name identification. 

The present invention preferably uses a buffer of 
81 92 bytes to improve transfer efficiency even though a 
buffer size of from 512 bytes to 8192, except for 4096 
for certain audio formats when the data is audio, will 
work. 

The present invention, through the network inter- 
face provides two methods of message deletion: imme- 
diate or marking messages to be deleted with a deletion 
"Commit" (see figure 8) at the end of a session. The sec- 
ond option is like the deletion feature of the audio (tele- 
phone) interface where messages to be deleted can be 
listened to again thereby removing the delete flag and . 
only those messages flagged for deletion are deleted 
when the audio session is ended. 

The present invention also allows different types of ■ 
messages to be bundled together into a multimedia type 
message with multiple body parts. The state information 
for a message includes body part information which in- 
dicates the content type of the body part. 

The present invention is also suitable for providing 
electronic data interchange (EDI) services where EDI 
forms, such as purchase order forms, are provided to a 
user for the purchase of goods, etc. Other types of data 
such as weather data can also be stored and transmit- 
ted. 

The present invention, using the administration fea- 
tures, can be configured through the network interface 
to perform operations such as sending standard text or 
voice messages to doctor's patients. - 

The administration of mailbox features, such as the 
password, telephone ring count, etc., is performed using 
HTML templates that can be customized for each serv- 
ice provider. 

The present invention provides priority of access to 
a mailbox by one of the owners to accesses that are 
made through the telephone interface. 

The browser 144, if it automatically requests a re- 
fresh ol a currently displayed page, allows a page to be 
created that includes a message list icon that can be * 
updated to reflect that a new message has arrived dur- " 
ing the session. 

The present invention also includes an automatic 
log-off feature that will log a subscriber out of the system .: 
when there has been no activity for a period of time. This 
allows subscribers to inadvertently leave their PC 
logged in to the system, such as when going home from 
work, and prevent others from accessing the system 
during the absence. 



The administration features of the system allows a 
■ verified system administrator to access a system admin- 
istration home page showing variables such as alarms, 
number of users logged in, etc. , and to perform functions 
s such as releasing an access block on a subscriber mail- 
box. ' 

The present invention has been described with re- 
spect to handling text messages such as e-mail and fac- 
simile. The text messages could have other formats 
io such as a word processing format, a spread sheet for- 
mat, a database format, etc. and could be other MIME 
type information than can be retrieved without conver- 
sion. The applications that are performed by the appli- 
cation processing units could include personal informa- 
15 tion managers, appointments, address bocks, etc. 

The many features and advantages of the invention 
are apparent from the detailed specification and, thus, 
it is intended by the appended claims to cover all such 
features and advantages of the invention which fall with- 
20 j n the true spirit and scope ol the invention. Further, 
since numerous modifications and changes will readily 
: > occur to those skilled in the art, it is not desired to limit 
. : the invention to the exact construction and operation il- 
lustrated and described, and accordingly all suitable 
2S modifications and equivalents may be resorted to, falling 
within the scope of the invention. 

Claims i 

30 

1. A message storage system, comprising: 

a multimedia mailbox storing voice messages 
and text messages; '■■ 
35 a voice interface providing access to the mes- 

sages via a telephone; and 
a network interface providing access to the 
messages over a network via a personal com- 
puter. 

40 

2. A system as recited in claim I , wherein said network 
interface comprises an Internet server and said 
voice interface comprises a voicemail server. 

"5 3. A system as recited in claim 2, wherein said network 
interface provides a home page and an access by 
a user of the home page by &n Internet browser gets 
an audio voice message when a the user selects a 
voice message and gets an image when the user 

so selects a text message. 

4. A system as recited in claim 3, wherein said home 
page is refreshed each time it is accessed. 

•55 5. a system as recited in claim 4, wherein said home 
page includes an expired expiration date. 

6. a system as recited in claim 1, wherein said mailbox 
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stores still and motion video messages. 

7. A system as recited in claim' 4, wherein said network 
interface streams the messages to the user. 

8. A system as recited in clairh 7, further comprising a 
personal computer including a prowser which plays/ 
displays messages streamed to the user. 

v. 

9. A system as recited in claim 1 , wherein said voice 
interface outputs the voice and text messages as 
audio to said telephone. 

1 0. A system as recited in claim 1 , wherein the network 
interface includes a data conversion cache for stor- 
ing converted messages. ' 

11. A system as recited in claim 1, wherein said network 
interface provides for recording messages. 

12. A system as recited in claim 1, wheiein message 
identifiers include a session number and a random- 
ly assigned identifier. 

13. An apparatus, comprising:! 

a message storage system including a multi- 
media mailbox storing voice messages and text 
messages, providing access to the messages via a 
telephone, and providing access to the messages 
via a personal computer. 

14. A message storage and retrieval system, compris- 
ing: ; 
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sage list including message identifiers compris- 
ing a session number and a randomly assigned 
file identifier responsive to a home page re- 
quest by the browser, retrieving the messages 
from the processing unit and streaming the 
messages to the computer responsive to 
browser message play requests to the home 
page having an expired expiration date, the 
computer playing an audio of the voice mes- 
sages in realtime as the voice messages are 
received, displaying an image of the text mes- 
" sages in real-time as the text messages are re- 
: ceived and displaying images of video messag- 
es in real-time as the video messages are re- 
ceived, and said computer recording a mes- 
. sage and forwarding a message to said net- 
; work unit for storage in said processing unit. 

15. A storage media, comprising: 

a multimedia mailbox storing voice messages 
and text messages, a process providing access to 
the messages via a telephone, and a process pro- 
viding access to the messages via a personal com- 
puter. 

16. A method, comprising: 

providing access to messages in a multimedia 
mailbox storing voice messages and text mes- 
sages via a telephone; and 
providing access to the messages via a person- 
al computer. 
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a telephone; ) 35 

a telephone network coupled to said telephone; 
a computer including a digital network browser; 
a digital network couplable to the computer: 
and 

a distributed architecture message system cou- 
pled to sad telephone ^etwork and said digital 
network, said message system comprising: 
a digital switch coupled to said telephone net- 
work; 

a control unit storing addresses of multimedia 
messages including voice, text, and video mes- 
sages, and controlling switching of said digital 
switch; j : 

a processing unit coupled to said digital switch, 
storing and retrieving the multimedia messag- so 
es, and outputing the voice and text messages 
to said telephone as-audio over said telephone 
network responsive toj telephone commands; 
a local network couple'd to said control unit and 
said processing unit; and ss 
a network unit coupled to the digital network 
and said local network! including a data conver- 
sion cache, providing a home page with a mes- 
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(54) A system for accessing multimedia mailboxes and messages over the internet and via 
telephone 



(57) A unified messaging system that provides a 
multimedia mailbox. The system allows a subscriber to 
access scored multimedia messages, such as voicemail 
messages, facsimile messages, combined voice and 
facsimile messages and video messages, not only 
through a public switched telephone network using a te I- 
ephone but also over a data network, such as the Inter- 
net or an intranet, using a personal computer. The sys- 
tem provides voicemail access over the telephone net- 
work., indicating message number, etc. with the ability to 
play messages to the telephone user as desired. For 
text type messages, such as facsimile and e-mail, the 
system converts the text inlo[speech and plays the 
speech to the telephone user. The system allows a per- 
sonal computer user to obtain the data network access 
using an Internet browser. The browser is used to ac- 
cess a home page of the system and get information 
about the messages stored, and is used to download 
(get) and play the messages at the personal computer 
via data streaming in the case of a voice or video mes- 
sages or view the messages \p the case of text type 
messages, such as facsimile and e-mait. The user can 
also perform the other typical messaging functions over' 
the data network connection that are provided for tele- 
phone access, such as viewing a message list, saving 
and deleting messages, group list administration and 



other administration tasks. 
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